Shares

A couple of weeks ago OpenEvidence dropped into my feeds. I am, as is no surprise to anyone who reads this blog, a fan of evidence. And I like open. Open skies, open vistas, open source. I use Linux as my daily driver and write most of my screeds in Neovim. So open and evidence are two great tastes that should be great together, to coin a phrase.

What is OpenEvidence? The front page calls themselves

The leading AI-powered medical information platform.

Leading, mind you. Well, I may not be enamored of AI, but I can use adjectives with the best of them. As the leading SBM blogger, well leading elderly blogger who is also a retired an infectious disease doctor, I feel particularly qualified to see what they have to offer, even if I may not be

Trusted by medical professionals at 7,000+ care centers across the US.

but I do have more than passing knowledge on SCAMs and infectious diseases. I may not have hoovered the internet, but I have read thousands of articles over the years on those topics. I would be confident go in against an AI Sicilian when medical knowledge is on the line.

While they have Open in their name, they do not explicitly mention if they have trained their AI ethically (i.e., respecting copyright) or it they scraped the internet for content, the current euphemism for stealing other people’s work. There were references behind paywalls, so I wonder how the papers were accessed. They have partnered with Elsevier, so it looks like they are training their AI on the up and up.

I do agree that

Information overload is an extreme challenge in medicine.

The quantity of medical information is staggering. There are easily 20,000 articles a year on PubMed categorized as infection. I would have to read 55 a day to read them all. My ID attending as a medical student called reading the medical literature as drinking from a fire hose. I am not certain why medical articles are referred to as literature. They ain’t no Great Gatsby.

The number of quality medical studies that are relevant to a given practice? Not so much. For my infectious disease podcast I looked at 1000 titles a month, downloaded 100, read 50 in detail and talked about 25 articles. I bet I easily read 50,000 articles since medical school. To my mind the medical literature was more about finding the needle of relevance in the haystack of worthless.

I am not so sure the solution is

To tame the medical information firehose, we built OpenEvidence, an artificial intelligence system to aggregate, synthesize, and visualize clinically relevant evidence in understandable, accessible formats that can be used to make more evidenced-based decisions and improve patient outcomes.

I am old school. I think you have to actually read the original literature and over a lifetime time, slowly and painfully understand medicine. The Readers Digest version ain’t gonna do it if you want to master a topic. You don’t get good at killing robots with a sword by playing Nier: Automata, you have to go out and kill them for reals.

I thought it would be interesting to see what OpenEvidence had to say about some common SCAMs. SCAMs are tricky, since there are many pitfalls in applying the standards of EBM to fiction based medicine. There are so many poorly designed studies, biases, and spin that goes unrecognized unless you lean to look for them. It is why there is SBM.

They do have their own version of the quack Miranda:

OpenEvidence is an experimental technology demonstrator. OpenEvidence does not provide medical advice, diagnosis or treatment.

Which is fair enough. Only a fool takes medical advice from the internet. So buyer beware.

I started with homeopathy, since of all the SCAMs homeopathy is the simplest to understand. Really. It is water. And, outside of thirst, there is no physiologic process homeopathy can affect. Unless you are either delusional (aka a practitioner) or totally clueless about reality.

I asked, “What is homeopathy effective for?” and received the following my annotation in bold:

Homeopathy is not widely accepted as an effective treatment for any specific medical condition within the mainstream medical community (you mean reality-based community?) The evidence supporting its efficacy is limited and often conflicting (because homeopathy is worthless).
Some studies suggest potential benefits in certain conditions, but these findings are not robust (weaselly understatement) enough to warrant widespread clinical use. For example, a review of clinical research indicated that homeopathy might have some effects in conditions such as allergic oculorhinitis, influenza-like syndromes, otitis, allergic complaints, fibromyalgia, sinusitis, rhinoconjunctivitis, and arthritis.[1] However, the evidence is often derived from studies with methodological limitations, and the results are inconsistent (**it begs the question as to why water would be effective any of the above. OpenEvidence isn’t trained to understand Bayes and prior probability. **).
In the field of psychiatry, a systematic review found some efficacy for homeopathy in functional somatic syndromes like fibromyalgia and chronic fatigue syndrome, but not for anxiety or stress.[2] Another meta-analysis reported mixed results for psychiatric disorders, with some studies showing efficacy in major depressive disorder and premenstrual syndrome, but overall, the data were insufficient to support clinical use.[3]
A systematic review on allergic rhinitis found that homeopathy might offer some symptom relief, but the overall quality of evidence was low, and the results were not conclusive.[4]
In summary, while some studies suggest that homeopathy may have effects in certain conditions, the overall quality of evidence is low, and there is no strong consensus supporting its efficacy for any specific medical condition. (True, but shows a complete lack of understanding about the basics of homeopathy) More rigorous, high-quality research is needed to draw definitive conclusions. (Nope. The book is closed about the efficacy of water on the treatment of diseases. If the I in AI actually stood for intelligence, it would ‘know’ that).

Three references to support the above, all meta-analyses. That is the synthesis of a vast literature (6800 hits on Pubmed) on the effects of water on a variety of diseases? Really. That is the best AI could do. It this was a med student talk (students on medical rotations are often assigned a topic to review and report at rounds the next day) I would assume they had their radiology residency and didn’t give a crap and made the barest of efforts to understand the topic. D minus.

So I asked, how does homeopathy work? Because it doesn’t.

Homeopathy is a therapeutic system based on the principle of “like cures like,” also known as the “Similia Principle.” This principle posits that substances that produce symptoms in healthy individuals can be used to treat similar symptoms in sick individuals.[1] Homeopathic remedies are prepared through serial dilution and succussion (vigorous shaking), which proponents claim imprints information into the water used for dilution.[2] (All of which is divorced from the reality. It is a dilution, er I mean delusion).

The pharmacodynamics of homeopathic remedies, particularly those in high dilutions, remain controversial. (Controversial? I think the AI is modeled on Vincini). Some hypotheses suggest that these remedies may work through mechanisms such as bioelectromagnetic information, water chain signaling, and regulation of systemic network bifurcation points.[3] (Really? You present this like these explanations have actual meaning). Another model proposes that homeopathic potentization involves the formation of stable nanodomains in water, which can retain and transfer electromagnetic information from the original substance.[4] (These ‘mechanisms’ are bullshit and tells you absolute nothing about how homeopathy would work if it could work. Such garbage out.)

I asked the question a week later and received a slightly different, although equally nonsensical answer:

Several hypotheses have been proposed to explain how homeopathy might work, including the formation of nanodomains in water, electromagnetic resonance, and quantum entanglement

Quantum entanglement? Seriously? I went to the reference and it says.

The quantum theory of ‘non-local’ phenomena inspires the idea of an ‘entanglement’ process among patient, practitioner and the remedy.

It is been many a year since my degree in physics. The third year of a physics BS is quantum mechanics. I still enjoy reading on the topic. I can recommend Quantum Drama, which I am currently reading.

It is argued that no one really understands quantum mechanics. I don’t. But I know what it isn’t, and it isn’t the above description. That sentence, and that reference, are gibberish. Neither homeopaths nor OpenEvidence can recognize it as such.

Next up? Chiropractic. I tried to ask the same question each time, so what is chiropractic effective for? And the answer is

Chiropractic care is primarily effective for managing musculoskeletal conditions, particularly low back pain and neck pain (Significant Vinzini? . Evidence supports its use in these contexts, showing that chiropractic care can provide significant pain relief and improve function. For instance, a study published in JAMA Network Open demonstrated that adding chiropractic care to usual medical care resulted in moderate short-term improvements in low back pain intensity and disability among active-duty military personnel.[1] (They say significant, whatever that means, but give as a single instance, moderate short-term improvements. Like a placebo. So somehow the entire literature can be summarized from one study? And that study?. Patients received more than just chiropractic:

Usual medical care included self-care, medications, physical therapy, and pain clinic referral. Chiropractic care included spinal manipulative therapy in the low back and adjacent regions and additional therapeutic procedures such as rehabilitative exercise, cryotherapy, superficial heat, and other manual therapies.

Color me unimpressed. If this had been human reporting the results, I would ask Dr. Vincini if they actually read the paper.)

Additionally, chiropractic care has shown potential benefits for other conditions. A systematic review indicated that chiropractic care might benefit patients with asthma, cervicogenic vertigo, and infantile colic. There is also promising evidence for its use in managing otitis media in children and pneumonia in elderly patients.[2] (There is no reason on earth that chiropractic would do diddly for any of the above. But, As an ID Doc I looked at the pneumonia article, what do they call it, an instance. Snicker. It was a systemic review. Again. So I looked at the two references for pneumonia. Both were osteopathic manipulations. Maybe, maybe not the same as chiropractic. When I think chiropractic, I think cracking the spine to adjust mythical subluxations. In the study the pneumonia patients received

The manipulation techniques of the OMT protocol were administered in the following sequence: thoracolumbar soft tissue, rib raising, doming of the diaphragm myofascial release, cervical spine soft tissue, suboccipital decompression, thoracic inlet myofascial release, thoracic lymphatic pump, and pedal lymphatic pump.

That ain’t chiropractic.

And the other study appears to the pilot study for the randomized study. The devil is always in the details, but I would not consider that study as evidence for chiropractic efficacy. But given the references, it is obvious that this AI is not really looking at the literature with anything that vaguely resembles an eye for the details).

For pregnancy-related low back pain and pelvic girdle pain, chiropractic care has been found to have inconclusive but favorable evidence (which is it? Favorable or inconclusive?), suggesting it may be beneficial for some patients (the weasel words are strong in this one – Yoda). Similarly, postpartum-related low back and pelvic girdle pain may also respond to chiropractic interventions, although the evidence is not definitive.[3-4]

In summary, chiropractic care is most effective for musculoskeletal pain, particularly low back and neck pain, with some evidence supporting its use for specific nonmusculoskeletal conditions. Further research is needed to solidify its role in these additional contexts. (Nope. No biological plausibility or quality studies suggesting efficacy, so no further research needed. Chiropractic, like homeopathy, is a closed book)

Again, 4 paragraphs and 4 references for a complex literature (9900 hits on Pubmed). Either systematic reviews or misleading primary literature. And if you do not read the primary literature, you will never understand the topic. So far OpenEvidence is not cutting the mustard. This ain’t Readers Digest, which looks encyclopedic in comparison.

The chiropractic subluxation complex is a fantasy. No such thing. What does OpenEvidence say?

A chiropractic subluxation complex, often referred to as a vertebral subluxation complex (VSC), is a concept in chiropractic medicine that describes a purported misalignment or dysfunction of the vertebrae that is believed to affect nerve function and overall health. The term encompasses a range of theories and models that have evolved over time.

The VSC is thought to involve biomechanical derangements of the spine that can lead to neurological disturbances. This concept has been central to chiropractic practice, serving as a primary indication for spinal manipulation therapy (SMT).[1] Theories around VSC suggest that these spinal misalignments can cause a variety of health issues by disrupting normal nerve function, although this remains a controversial and debated topic within both the chiropractic and broader medical communities.[2]

Historically, the first subluxation complex model was proposed by Faye, and subsequent models were developed by theorists such as Lantz and Dishman. These models were integrated into consensus statements by the International Chiropractic Association and the American Chiropractic Association.[3] Despite the historical and theoretical significance of VSC in chiropractic practice, there is a call for more empirical research to validate these concepts and their clinical implications.[3]

In summary, a chiropractic subluxation complex is a theoretical construct in chiropractic medicine that describes vertebral misalignments believed to affect nerve function and health, though its clinical validity remains a subject of ongoing debate and research.

Not bad for Dr. Vincini, if a bit too accepting. Again, no need for further research a fictional process.

Can chiropractic manipulation cause a stroke?

Yes, chiropractic manipulation can cause a stroke, specifically through the mechanism of cervical artery dissection (CAD). This association has been documented in multiple studies and case reports.

At least they got that right.

And lastly, our favorite theatrical placebo, acupuncture. What is acupuncture effective for? Hint: nothing. But OpenEvidence notes

Acupuncture has been shown to be effective for several conditions, particularly in the realm of pain management and certain cancer-related symptoms.

Pain Management: Acupuncture is effective for chronic pain conditions such as chronic low back pain, tension-type headaches, and myofascial pain syndrome. For chronic low back pain, acupuncture has demonstrated a clinically meaningful reduction in self-reported pain and improved function compared to sham acupuncture and no treatment. It is also effective in reducing the frequency and severity of chronic tension-type headaches. For myofascial pain syndrome, acupuncture significantly reduces pain intensity and improves pain-related outcomes.[1-4]

Cancer-Related Symptoms: Acupuncture is beneficial for managing various cancer-related conditions. It has shown efficacy in reducing cancer-related pain, fatigue, insomnia, and improving quality of life. It is also effective for managing chemotherapy-induced nausea and vomiting, bone marrow suppression, and specific symptoms related to breast cancer such as menopausal symptoms and hot flashes.[5]

Mental Health: Acupuncture, particularly when combined with pharmacological treatments, has shown effectiveness in treating major depressive disorder. Electro-acupuncture combined with SSRIs has been found to be more effective than either treatment alone.[6]

Functional Dyspepsia: Acupuncture has demonstrated positive effects on anxiety and depression in patients with functional dyspepsia, showing superiority over placebo and first-line drugs in reducing anxiety and depression scores.[7]

The American College of Physicians recommends acupuncture as a noninvasive treatment option for acute, subacute, and chronic low back pain, citing moderate improvement in pain relief and function.[8]

Really? This blog has many an article to refute all of the above. All the alleged effects of acupuncture can be nicely accounted for by poor study design, bias and spin. But 5 paragraphs and 8 references for a complex literature with 46,000 hits on Pubmed? Can that really do the topic justice? You learn nothing about a topic with this kind of summary. And the AI is equally unable to read the acupuncture mechanism literature with a discerning eye.

How does acupuncture work?

Acupuncture works through several mechanisms involving both peripheral and central nervous systems, as well as neuroimmune interactions.

Peripheral Mechanisms: Acupuncture stimulates specific points (acupoints) on the body, which activates high-density primary afferent fibers, including myelinated A fibers and unmyelinated C fibers. This activation leads to the release of various molecules such as adenosine, histamine, and β-endorphins at the local site, which interact with receptors on nerve endings and immune cells, modulating pain and inflammation.[1-3]

Central Mechanisms: The signals from the activated afferent fibers are transmitted to the spinal cord and brain. In the spinal cord, acupuncture modulates pain gating mechanisms, inhibits long-term potentiation (LTP) of the spinal dorsal horn, and reduces wide dynamic range (WDR) neuronal activities. At higher brain centers, acupuncture affects regions such as the medial prefrontal cortex, anterior cingulate cortex, and thalamus, which are involved in pain perception and modulation.[1][4-5]

Neuroimmune Crosstalk: Acupuncture also influences neuroimmune interactions. It can modulate the release of pro-inflammatory cytokines and other immune mediators, thereby reducing inflammation and pain. This involves pathways such as the HMGB1/RAGE and TLR4/NF-κB signaling pathways, and the activation of TRPV1 and TRPV2 channels on mast cells.[4][6]

Electroacupuncture (EA): EA, a combination of traditional acupuncture and electrical stimulation, has been shown to enhance these effects by more robustly activating A fibers and modulating both peripheral and central pain pathways.[1][7]

In summary, acupuncture works by activating peripheral afferent fibers, modulating central pain pathways, and influencing neuroimmune interactions to achieve analgesic and anti-inflammatory effects.

This is not quite an AI hallucination where there are incorrect or misleading results that AI models generate. Huh. When you look at the definition, it applies to the production of most of the SCAM literature. Acupuncture research as hallucination? Works for me. While OpenEvidence may be artificial, it sure ain’t intelligent. The AI is equally accepting of a variety of explanations for acupoints and meridians. Not being discerning, the AI has to accept all sorts of contradictory nonsense at face value. Wait. Maybe the AI is modeled on a naturopath. That would explain a lot.

You can ask similar questions of reiki or another SCAMs and get similar responses.

But understanding SCAMs is a learned skill. One cannot apply the standards of reality-based medicine to therapies that are based on fiction. Most docs can’t. It is reflected in the literature, which is reflected in this version of AI. When it comes to SCAM, AI is a gullible rube who just fell off the turnip truck. Because I asked it, what is science-based medicine? And it replied

Science-based medicine, often referred to as evidence-based medicine (EBM), is a systematic approach to clinical practice that emphasizes the use of the best available research evidence, combined with clinical expertise and patient values, to make decisions about the care of individual patients.

Nope. Not the same. And isn’t the part of point of EBM to get away from clinical expertise, aka the three most dangerous words in medicine? And while important in providing care, patient values have no bearing at all on the evidence or science outside of fictional entanglement. Steve? David? Gotta get the message out. Big AI fail here.

When asked if the MMR causes autism, it was nicely definitive in bold no less.

No, the measles, mumps, and rubella (MMR) vaccine does not cause autism.

And they had real references rather than the always problematic systemic review/meta-analysis.

I was struck at how a complex literature was reduced to a handful of paragraphs and, for SCAMs, very badly sourced references. Reminded me of my kids middle school English class where they focused on the 5-paragraph essay. It seems OpenEvidence did not progress beyond 8th grade.

Understanding the medical literature is hard and time consuming and there are not shortcuts. For SCAMs, OpenEvidence demonstrated incompetence. But there is real medicine. How would the AI do for that? So I picked a topic that is both common and often screwed up. How do you treat S. aureus bacteremia? And they got it right.

The treatment of Staphylococcus aureus bacteremia (SAB) involves several critical steps to optimize patient outcomes.

  1. Antibiotic Therapy: For methicillin-susceptible Staphylococcus aureus (MSSA) bacteremia, first-line treatment typically includes a beta-lactam antibiotic such as cefazolin or nafcillin. For methicillin-resistant Staphylococcus aureus (MRSA) bacteremia, vancomycin or daptomycin are the preferred agents. The American Heart Association recommends vancomycin for a minimum of 6 weeks for MRSA endocarditis, with or without gentamicin for the first 3 to 5 days.[1-4]
  2. Echocardiography: All patients with SAB should undergo echocardiography to rule out infective endocarditis. Transesophageal echocardiography (TEE) is preferred due to its higher sensitivity compared to transthoracic echocardiography (TTE).[3]
  3. Source Control: Identifying and controlling the source of infection is crucial. This may involve the removal of infected prosthetic material or drainage of abscesses.[1]
  4. Duration of Therapy: The duration of antibiotic therapy is typically 2 weeks for uncomplicated cases and 4-6 weeks for complicated cases, such as those with endocarditis or metastatic infections.[1][3]
  5. Infectious Diseases Consultation: An infectious diseases specialist should be consulted for all cases of SAB to guide management and improve outcomes.[1]
  6. Combination Therapy: For persistent MRSA bacteremia, combination therapy with agents such as daptomycin and ceftaroline may be considered, although high-level evidence is still needed.[2][5]

In summary, the management of SAB involves appropriate antibiotic therapy, echocardiography, source control, determining the duration of therapy, and infectious diseases consultation. These steps are supported by clinical guidelines and literature to optimize patient outcomes.

I got no quibbles with this. It is a correct, if very superficial, summary. Again, they only had 5 reviews as references. Hardly a summary of the literature (8000 Pubmed hits), but a start. The rationale for why you do the above is not mentioned, and if you do not know the why of it, you do not know what you are doing.

I, for one, am glad they recommend ‘appropriate’ antibiotic therapy. So much better than the alternative. I have to wonder who would want to ask this question and find the reply useful. If a student or resident, way too simple to understand the topic. If an ID specialist, it is a summary courtesy of Captain Obvious. And if you are, say, a hospitalist, if you are using this as your guide, I would wonder if you should have patient care responsibilities.

I asked some other questions on common infectious diseases (UTI, cellulitis, pneumonia) and always got 5 paragraph replies with 3 or 4 references from mostly guidelines and systematic reviews. I can say the results were correct, but so lacking in context and understanding I have trouble envisioning a target audience. I suppose a health care provider who wants minimal understanding of a topic? A summary of a summary? I’ll pass. It might be trusted by medical professionals at 7,000+ care centers across the US, but I would be skeptical of trusting a medical professional who relied on this for their understanding. There are no shortcuts to mastering the medical literature.

The SCAM blind spot of evidence-based medicine is alive and well at OpenEvidence. They need to add SBM programming to their AI model. I wonder if that blind spot is amenable to remediation? I hope they have the concept of a plan to incorporate Science-Based Medicine into their AI. I doubt it it is doable. Often can’t get Real Intelligence to do it.

Photo: “Artificial Intelligence & AI & Machine Learning” by mikemacmarketing is licensed under CC BY 2.0 !.

Shares

Author

  • Mark Crislip, MD has been a practicing Infectious Disease specialist in Portland, Oregon, from 1990 to 2023. He has been voted a US News and World Report best US doctor, best ID doctor in Portland Magazine multiple times, has multiple teaching awards and, most importantly,  the ‘Attending Most Likely To Tell It Like It Is’ by the medical residents at his hospital. His multi-media empire can be found at edgydoc.com.

    View all posts

Posted by Mark Crislip

Mark Crislip, MD has been a practicing Infectious Disease specialist in Portland, Oregon, from 1990 to 2023. He has been voted a US News and World Report best US doctor, best ID doctor in Portland Magazine multiple times, has multiple teaching awards and, most importantly,  the ‘Attending Most Likely To Tell It Like It Is’ by the medical residents at his hospital. His multi-media empire can be found at edgydoc.com.